Tripoli has condemned the UK for recognising the rebels as Libya's "sole governmental authority" after similar moves by France and the US.
Khaled Kaim, deputy foreign minister in Muammar Gaddafi's government, told reporters the decision was unprecedented and irresponsible.
Libya would seek to reverse the decision through the courts, he said.
Britain has ordered the expulsion of all eight remaining Gaddafi diplomats in the UK.
The rebel leadership, the National Transitional Council (NTC), has put forward Mahmud al-Naku, a writer and journalist, as the new Libyan ambassador in London.
He told the BBC he has been in exile for 33 years because of his opposition to the Gaddafi regime.
Foreign Secretary William Hague said the NTC had shown its commitment to a "more open and democratic Libya... in stark contrast to Gaddafi whose brutality against the Libyan people [had] stripped him of all legitimacy".
The green flag of the Gaddafi government was still flying outside the Libyan embassy in Knightsbridge on Wednesday afternoon as protesters carrying the red, green and black flag of the rebels gathered outside.
Folllowing the US decision to recognise the NTC two weeks ago, the BBC has been told the US has now received an "official request" from the rebels to reopen the Libyan embassy in Washington.
US officials say they are "reviewing" the request.
Libyan rebels and pro-Gaddafi forces are still locked in battle, five months after an uprising began against Muammar Gaddafi's 42-year rule, as Nato continues to enforce a UN-backed no-fly zone over the country.
'Surprise decision'
Mr Kaim said the British decision was "unprecedented in diplomatic history".
"It's illegal, it's irresponsible and for us it was a surprise to happen from the British Government because, I mean, if other countries will [follow] Britain, then the international diplomacy will be chaos," he told reporters in Tripoli.
"Of course it both violates national British laws and international law. Specifically the Vienna convention on diplomatic relations of 1961," he said.
"I personally consider it a stain on the forehead of Britain," he added. Libya, he added, would try to reverse the decision by taking legal action in both British courts and the International Court of Justice.
The deputy foreign minister also condemned Britain and France for supporting the rebels in eastern Libya. They were, he said, "flogging a dead horse".
Col Gaddafi's charge d'affaires, Khaled Benshaban, is being given three days to leave while his colleagues' fate will be decided on a "case-by-case basis", the Foreign Office said.
Mr Hague told reporters in London that a recent meeting of the Libya contact group in Istanbul had decided to treat the NTC as the legitimate government authority in Libya - and he was outlining the UK's response.
The UK previously said it recognised "countries not governments" but the British foreign secretary argued it was a "unique situation" and recognising the NTC could help "legally in the unfreezing of some assets".
The British government will now be able to unfreeze £91m ($149m, 102m euros) in assets from a Libyan oil company but not other Libyan assets frozen in the UK, which total about £12bn.
Libya's rebels have been saying for weeks that they need more money from the international community to pay for salaries and food supplies.
In the rebel stronghold Benghazi, NTC chairman Mustafa Abdul-Jalil praised the British decision as an economic and political boost.
The new ambassador to London, Mahmoud al-Naku, told the BBC Col Gaddafi could not stay in Libya once he was overthrown.
"Thanks to the brave men on the ground who now surround the main cities, we hope in a few weeks [the rebels] will arrive to Tripoli," he said.
Meanwhile, outside the embassy in London, about 20 demonstrators chanted, "We want to say thank you UK".
"I was so happy I cried," one of them, 57-year-old Muftah Abdelsamad, told AFP news agency.
"We have been wanting this for six months now. It will put pressure on Gaddafi to leave the country because there is no other way for him. Soon we will have freedom and democracy in Libya."
Source: BBC
